Aesthetic imaging system

ABSTRACT

Disclosed is an aesthetic imaging system (20) for use in editing digital images. The aesthetic imaging system includes an imaging program (21) that runs on a personal computer (28) having an image capture board (30), a monitor (32), a video source (34) for providing digital images to be edited by the aesthetic imaging system, and a pen and tablet 38 for use in editing the images. The imaging program includes a unique combination draw tool that includes a freehand draw mode, a curve mode and an undo mode that are available without cycling through menus. The combination draw tool may be used with any of the draw tools. Another feature of the imaging program is autoblend, a rectangular user interface that is invoked by each of the shape tools. The autoblend interface simplifies editing when using shape tools by consolidating the move, paste and blend, and paste without blending commands into a single, convenient interface.

RELATIONSHIP TO OTHER APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 08/406,201, filed Mar. 17, 1995, now U.S. Pat. No. 5,687,259,the benefit of the filing of which is hereby claimed under 35 U.S.C.§120.

FIELD OF THE INVENTION

This invention generally relates to computer imaging programs and, morespecifically, to a method and apparatus for manipulating digitalphotographs.

BACKGROUND OF THE INVENTION

The digital age continues to present additional opportunities for visualcommunication using computers. As an example, digital photographs areroutinely being manipulated to produce a desired effect or result in themagazine and film-making industries. In the medical field,computer-based imaging has and continues to gain acceptance in aclinical setting as a viable communications tool between plastic or"cosmetic" surgeons and potential patients.

People are with increasing frequency consulting physicians aboutcosmetic surgery. While in many cases the patients considering cosmeticsurgery have an impressive understanding of the procedures available andmedical terms used to describe these procedures, it is apparent that theslightest miscommunication may result in dire consequences. This haspromoted the use of computer imaging to facilitate communication betweenthe physician and prospective patient. Specifically, high-end aestheticimaging systems allow a physician to take pre-operative digital imagesof the patient, e.g., including profile and frontal views. The imagesare stored in memory in the computer where they can then be edited.Using feedback from the patient, the edited images are useful in fullyunderstanding the procedures desired. The visual support provided by acomputer-based imaging system is extremely valuable on both sides. Acosmetic surgeon can more readily understand what patients hope toachieve by a cosmetic surgical procedure; and patients can view adetailed visual representation of predicted results, including both thebenefits and limitations of the procedure.

Existing aesthetic imaging systems typically use a number of tools toallow a physician to manipulate a pre-operative image of a patient toillustrate an intended post-operative result. Preferably, the toolsallow the physician to manipulate the preoperative image during aconsultation with a patient. By manipulating the image with the patientin attendance, the patient receives immediate feedback from thedisplayed results. For a successful preoperative consultation, the useof the editing tools should be as unobtrusive as possible. During theconsultation, a physician would like the patient to focus on the endresults of the surgery, rather than the technologic wizardry used todemonstrate those results on the video monitor. Unfortunately, theediting tools used in existing aesthetic imaging systems typicallyhinder rather than help the physician in demonstrating the results thatmay be achieved through cosmetic surgery.

Among others, a disadvantage of existing aesthetic imaging programs isthat a physician or facilitator in a pre-operative consultationtypically must go back and forth through many windows-based menus inorder to edit an image. Cycling between the various menus to invoke thetools necessary for a consultation is disadvantageous in that it is timeconsuming. For example, some physicians schedule a follow-up visits forpatients to allow the physician time to edit the images. More important,however, is that the process is distracting to the patient and tends tomake the pre-operative consultation all the more mystifying. As a resultof the disadvantages associated with prior art systems, some patientslose interest or become frustrated with the interview, both of which mayreflect back on the physician.

A further disadvantage of existing aesthetic imaging systems is that itis impossible for a physician or facilitator to display differentcombinations of the edits that they have performed. In existingaesthetic imaging programs, as a physician edits a patient's image, thephysician's edits are added to the preexisting edits of the image. Mostprograms are only capable of showing two version of the patient's image;the unedited, original version, and the final edited versionincorporating all of the physician's changes. It is therefore difficultfor the physician to show various combinations of the edits that hadbeen performed. For example, a physician may edit an image to removewrinkles around a patient's eyes and to narrow the patient's nose.Existing aesthetic imaging programs only allowed the physician tosimultaneously show all of these changes. If, for example, the patientwished to view the changes to the eyes without the changes to the nose,it was difficult for the physician to easily separate out the sequenceof edits that had been performed to adjust the patient's nose. Animproved aesthetic imaging system in which a physician can more easilyedit pre-operative images in response to a patient's suggestions andinquiries would be extremely advantageous.

A still further disadvantage of existing aesthetic imaging systems isthat the systems allow a physician to perform nearly flawless editing ofa patient's image. Unfortunately, the edits performed by a physician onan aesthetic imaging system are often unobtainable results that cannotbe achieved when actual surgery is performed. Unless the physician isespecially skilled at using the aesthetic imaging system, it isdifficult to show the patient achievable results, which typically fallwithin a range somewhere between the original patient image, and theoptimum results as displayed by the edited image on the screen. Ittherefore would be advantageous to develop an aesthetic imaging systemthat allowed a physician to display more realistic results that areachievable through surgery.

SUMMARY OF THE INVENTION

The invention is an aesthetic imaging system for use in editing digitalimages. The aesthetic imaging system includes a unique user interfacethat allows edits to be performed more efficiently and with lessconfusion to the patient.

In one aspect of the invention, a method of editing a digital imagecomprised of a plurality of color pixels in an aesthetic imaging systemis disclosed. The aesthetic imaging system including a processor, amemory, a monitor, and a pen and cooperating tablet for controlling acursor displayed on the monitor. The pen has a depressable tip and aside button, each of which include an on status and an off status,wherein the position of the pen tip relative to the tablet determinesthe position of the cursor on the monitor. The method comprises: (a)evaluating the following variables: (i) the status of the tip of thepen; (ii) the status of the side button on the pen; and (iii) movementof the pen tip relative to the tablet; (b) actuating a freehand drawingmode if a first set of variables are present, wherein movement of thepen relative to the tablet edits pixels that are located at positionscorresponding to the position of the cursor; and (c) actuating a curvedrawing mode if a second set of variables are present, wherein a linesegment is displayed between two endpoints and movement of the penrelative to the tablet stretches the line segment, forming a curve andediting pixels that are located at positions corresponding to theposition of the curve.

In another aspect of the invention, method further includes: (a)actuating a freehand undo mode if a third set of variables are present,wherein movement of the pen relative to the tablet restores pixels thatare located at positions corresponding to the position of the cursor totheir pre-edited color; and (b) actuating a curve undo mode if a fourthset of variables are present, wherein a line segment is displayedbetween two endpoints and movement of the pen relative to the tabletstretches the line segment, forming a curve and restoring pixels thatare located at positions corresponding to the position of the curve totheir pre-edited color.

In other aspects of the invention, the freehand draw mode is actuated ifthe tip of the pen is depressed and pressure is maintained while the tipis moved a predetermined distance. In another aspect, the curve drawmode is actuated if the tip of the pen is depressed and released withina predetermined distance.

In a further aspect of the invention, the curve draw mode is actuatedby: (a) establishing a first endpoint at the position of the pen whenthe second set of variables are present; and (b) monitoring the statusof the tip of the pen and establishing a second endpoint at the positionof the pen if the tip is toggled from an off state to an on state.

In another aspect of the invention, an improved prioritize feature isdescribed for viewing an image. A user may identify several areas in amodified patient image containing edits that alter the image from theoriginal image. As each area is identified by the user, an identifyingtag is assigned to each of the areas. When desiring to show variouscombinations of the edits that have been performed on the image, theuser may select the areas to display using the identifying tags. A usermay therefore quickly cycle through various permutations of theprocedures that have been edited for patient display.

In still another aspect of the invention, an improved user interface isprovided to minimize the distraction of a patient as the patient iswatching the image being edited. Preferably, a menu bar on the top ofthe display is removed during most editing, so that only the image ofthe patient is displayed. When the menu bar must be displayed, the baritself is transparent to allow the patient to see the image through themenu bar. Only the commands and the outline of the menu bar arepresented in a contrasting color, minimizing the overall visualimpression created by the menu bar.

In yet another aspect of the invention, a warp tool is described thatallows a user to quickly and easily manipulate various features in animage. To use the warp tool, a user first encircles a portion of theimage to be edited. Once the area has been selected, the user may tipthe pen to designate a stretch point within the selected area. As theuser floats the pen over the tablet, the image is then stretched as ifpulled from the stretch point. Areas of the image in the direction ofstretch are compressed, and areas away from the direction of stretch areexpanded. Areas surrounding the warping area are automatically adjustedto ensure that there is no discontinuities with the warping area. Themanipulation of the image is performed in real-time, allowing a patientto see the warping as it is being performed by a user.

An advantage of the tools and features described herein are that theyimprove the overall experience of a patient during a preoperative visitwith a physician. The powerful tools in the aesthetic imaging systemallow the physician to easily manipulate the patient's image in responseto feedback provided by the patient. The aesthetic imaging systeminterface also allows the patient to focus on the image beingmanipulated, rather than on the aspects of the aesthetic imaging systemthat allow the manipulation. The end result is an improved preoperativevisit that provides a more realistic impression of the results that aphysician may achieve through surgery.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same becomesbetter understood by reference to the following detailed description,when taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a block diagram of an aesthetic imaging system in accordancewith the invention;

FIG. 2 is a block diagram illustrating various buffers used by theaesthetic imaging system to store and manipulate data;

FIG. 3 is a flow chart illustrating an exemplary routine by whichdigital images may be viewed and edited using the aesthetic imagingsystem;

FIG. 4A is a flow chart of an exemplary routine for photographingpatients in accordance with the invention;

FIG. 4B is a flow chart of an exemplary routine for calculating achecksum value and comparing the calculated value to a previously storedvalue to determine if an image has been altered;

FIG. 5 is a flow diagram of an exemplary routine for implementing acombination tool for use with various drawing (draw) tools in accordancewith the invention;

FIG. 6 is a pictorial representation of an image to be edited;

FIGS. 7A-7E are pictorial representations of editing an image using aprior art imaging program;

FIGS. 8A-8E are pictorial representations of using the aesthetic imagingsystem to accomplish the identical edits shown in FIGS. 7A-7E;

FIG. 9A is a flow chart of an exemplary routine of a contour tool inaccordance with the invention;

FIGS. 9B-9C are pictorial representations illustrating the function ofthe contour tool of FIG. 9A;

FIGS. 9D-9G are pictorial representations illustrating exemplary editsthat may be accomplished using the contour tool;

FIG. 10 is a flow chart of an exemplary routine for implementing anautoblend tool in accordance with the invention;

FIG. 11 is a pictorial representation of a user interface forimplementing the autoblend tool of FIG. 10;

FIG. 12A is a flow chart of an exemplary routine illustrating a cutouttool in accordance with the invention;

FIG. 12B is a flow diagram of an exemplary routine illustrating a rotatetool in accordance with the invention;

FIG. 13 is a flow chart of an exemplary routine for viewing images inaccordance with the invention;

FIGS. 14A-14D are pictorial representations illustrating the effects ofa compare feature in accordance with the invention;

FIGS. 15A-15C illustrate a split image option of viewing images inaccordance with the invention;

FIG. 16 is a pictorial representation illustrating the use of atranslucent image to allow a patient to accurately position themselvesin order to capture a second image having the same location andorientation as an original stored image;

FIG. 17 is a pictorial representation illustrating a compare imagewherein a presurgical image of a patient is compared side-by-side with apostsurgical image having the same location and orientation;

FIGS. 18A-18C are pictorial representations illustrating the use of awarp shape tool to edit a patient's image;

FIGS. 19A-19E are pictorial representations illustrating the function ofthe warp tool;

FIG. 20 is a pictorial representation illustrating the use of atransparent menu bar when viewing an image of a patient; and

FIG. 21 is a flow chart of an exemplary routine for implementing a zoomviewing feature in accordance with the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

An aesthetic imaging system 20 in accordance with the invention isillustrated in FIG. 1. The aesthetic imaging system 20 includes animaging program 21 that runs on a processing unit 22 controlled by anoperating system 24. A memory 26 is connected to the processing unit andgenerally comprises, for example, random access memory (RAM), read onlymemory (ROM), and magnetic storage media such as a hard drive, floppydisk, or magnetic tape. The processing unit and memory are typicallyhoused within a personal computer 28 which may be, for example, aMacintosh™, International Business Machines (IBM™) or IBM-compatiblepersonal computer. When used with IBM and IBM-compatible personalcomputers, the operating system 24 may be DOS based or may incorporate awindowing environment such as Microsoft Windows™ or OS/2™.

The aesthetic imaging system also includes an image capture board 30that is coupled to the processing unit 22, a monitor 32, video source34, and printer 36. The video source, monitor, and printer are coupledto the processing unit 22 through the image capture board 30. The videosource may include one or more video cameras, a VCR, a scanner, orsimilar source for providing digital images to be edited by theaesthetic imaging system. The aesthetic imaging system further includesa pointing device, which is preferably a stylus (pen) and tablet 38,that is connected to the processing unit 22. In addition, the aestheticimaging system may include a modem 40 to provide on-line capabilities tousers of the system, such as technical support and teleconferencing.

The image capture board 30 has a plurality of buffers in high-speedmemory, e.g., RAM, that are used by the imaging program 21 to providevery fast response times to image edits. With reference to FIG. 2, fourbuffers are illustrated for use in explaining the operation of theaesthetic imaging system. These include an original image buffer 50, amodified image buffer 52, a current image buffer 54, and a workingbuffer 56. Suitable image capture boards for use in the aestheticimaging system include the Targa +64 and Targa 2000 boards, distributedby Truevision, Inc. of Indianapolis, Ind. The buffers are discussed inregard to a single pose only, such as a profile or front view of aperson.

The original image buffer 50 contains an unedited digital image, forexample, a side profile picture of a potential patient. The modifiedimage buffer 52 contains any edits made to a copy of the original image.The modified image buffer is updated during a save and after eachsession. The current image buffer 54 contains information identical tothe modified image buffer upon beginning a session. Thereafter, editsmade to the current image are saved in the working buffer 56 as anoverlay to the current image. During a save, the contents of the currentimage buffer 54 are copied to the modified image buffer 52, and theworking buffer 56 is cleared.

Prior to discussing the aesthetic imaging system in further detail, acompendium of terms used in the application may be helpful:

    ______________________________________                                        Image    A digital photograph or picture of a patient.                        Stylus   The "pen" that may be used to select menus, modify                            images, and carry out other commands in the                                   program. The stylus controls the cursor, just as a                            mouse pointing device does on a personal computer.                   Tablet, or                                                                             The electronic notepad used in conjunction with a                    Pad      stylus. The pen must be held relatively close to the                          pad in order for the pen to communicate with the                              tablet. Unlike a mouse, the tablet follows an X/Y                             grid that matches the monitor, i.e., if the pen is                            positioned at the top left corner of the tablet, the                          cursor is displayed at the top left corner on the                             monitor.                                                             Floating Moving the pen to move the cursor, without actually                           touching the tablet.                                                 Selecting                                                                              Selecting (also referred to as "tipping" or "pressing")                       the tip of the pen briefly onto the tablet. This selects                      a command or affects the drawing tool, depending on                           the current procedure being implemented.                             Cancel   Using the side button on the pen to execute a                                 command.                                                             Moving   Pressing the tip of the pen on the tablet, releasing it,                      then moving it across the tablet.                                    Pressing or                                                                            Pressing the tip of the pen, then dragging it across the             Tipping &                                                                              tablet while maintaining pressure. When using                        Dragging drawing and shaping tools, this turns the cursor into a                       drawing tool, enabling the user to draw freehand                              objects or "brush" the image in any manner.                          ______________________________________                                    

FIG. 3 illustrates an exemplary routine for implementing the imagingprogram 21 in accordance with the invention. At block 60, a systemstartup is performed wherein the computer looks for peripheral devicesthat are connected to the aesthetic imaging system the memory is tested,and any other startup procedures needed to get the system up and runningare implemented. At block 62, the imaging program displays a main menu,which provides access to the various features of the imaging program.Specifically, the main menu includes the following options: Storage,Camera, View, Shape, Draw, Print, And Exit. Those options that arepertinent to the invention are described in further detail below.

At block 64, a test is made to determine if the Camera option has beenselected from the main menu, indicating that the user wants to take apicture of a patient. If the Camera option has been selected, a routineto implement this command is called at block 66. A suitable subroutinefor this task in illustrated in FIG. 4. Upon return from the Cameraroutine, the program loops to block 62.

If the Camera option was not selected, a test is made at block 68 todetermine if the Storage option has been selected for the main menu,indicating that the user wants to load an image (picture) from computerstorage, e.g., a hard drive. If the Storage option was selected, theimage(s) to be edited or viewed during the current session are selectedat block 70.

At block 72, the selected images are copied to the appropriate buffersin the frame grabbing board, as described in FIG. 2 and the accompanyingtext. For example, if the selected image is an original image that hasnot yet been edited, the original image will be copied to the original,modified, and current image buffers. If the selected image is an imagethat has previously been modified, the original image is copied to theoriginal image buffer 50 and the modified image is copied to both themodified and current image buffers 52 and 54. It will be appreciatedthat the number of images that may be loaded at one time will belimited, in part, by the capacity of the frame grabbing board. Theprogram then loops to block 62.

If the Storage option was not selected, a test is made at block 74 todetermine if the Draw option has been selected from the main menu. Ifthe Draw option has been selected, a draw tool routine is called atblock 76. The program then loops to block 62. A suitable routine forimplementing the Draw option is shown in FIG. 5.

If the Draw option was not selected, a test is made at block 78 todetermine if the Shape option has been selected. If the Shape option hasbeen selected, a shape tool routine is invoked at block 82. A suitableroutine for implementing the Shape option is shown in FIG. 10.Otherwise, at block 82 a test is made to determine if the View optionhas been selected.

If the View option has been selected, a view subroutine is invoked atblock 84. An appropriate routine for the View option is shown in FIG.13. The program then loops to block 62. At block 86 a test is made todetermine whether the Exit option has been selected from the main menu.If not, the program loops to block 62. Otherwise, any edits to the imageare saved at block 88. At this point in the program, the image in thecurrent image buffer 54 is saved to the modified image buffer 52, andthe working buffer 56 is cleared. The program then terminates.

Taking Pictures of a Patient Using an Inverse Image

FIG. 4A illustrates an exemplary user interface that utilizes a videocamera for acquiring a digital image of a patient. It is noted a scanneror other input device may also be used to input an image into theaesthetic imaging system. In FIG. 4A, the solid blocks indicate userinterface options presented to the user by the aesthetic imaging systemand the dashed blocks represent system responses to the decisions made.At block 100, a patient is positioned in front of the video camera. In apreferred embodiment of the aesthetic imaging system, an inverse or"mirror" image of the patient's image will be displayed on the monitor,as indicated at block 101. The inverse image is computed using data fromthe original image, and is representative of how patients see themselvesday to day when looking into a mirror. More specifically, digital imagesare comprised of pixels or picture elements. It is known to thoseskilled in the art how digital image pixels may be manipulated to createan image that is the inverse of the original.

Displaying an inverse image of a patient is advantageous when takingpre- and post-surgical pictures of patients because it allows patientsto more easily center or otherwise position themselves on the monitor.Without the pixel manipulation, the input from a camera or other digitaldevice may create confusion when positioning a patient. Under normalviewing, if a patient appears left of center in the monitor, they are inreality too far to the right. In this instance, a typical patient'sreaction is to move even further to the right. With a mirror imagedisplayed, the tendency of most patients is to naturally adjust to thedesired position.

The displaying of a mirror image is particularly important when takingpost-surgical pictures. In post-surgical pictures, it is advantageous tohave the patient in exactly the position they were in when taking thepre-surgical picture. For post-surgical pictures, the aesthetic imagingsystem will preferably display a translucent inverse image of thepre-surgical picture on the monitor, and then overlay an inverse imageof the picture currently being taken. As shown in FIG. 16, a translucentpatient image 370, in this case a patient's profile, is displayed on themonitor. The translucent image is the preoperative image taken prior toundergoing a surgical procedure. A "live" video image 372 of the patientis also displayed under the translucent inverse image. By moving therelative positions of the patient and the camera, a user may positionthe patient in the identical orientation with which they took thepresurgical picture. When the patient is appropriately positioned, thetip of the pen may be pressed anywhere on the tablet to freeze thedigital image on the monitor. Patients can thus easily align themselveswith their former picture to achieve very similar before and afterpictures.

The advantage of the translucent method of aligning a patient is shownin the side-by-side display shown in FIG. 17. As shown in FIG. 17, theaesthetic imaging system of the present invention may generate aside-by-side display of the two images to allow a patient to easily andaccurately compare a presurgical picture with a post-surgical picture.The left half of the monitor may display the presurgical image 370, andthe right half of the monitor may display the post-surgical image 372.Allowing a patient to view the two images side-by-side in precisely thesame orientation provides the patient with an accurate impression of theresults achieved by surgery.

Upon establishing a desired position for the patient, the image isfocused and sized at block 102 by using the aesthetic imaging system toadjust the electronic controls on the video camera. After anyadjustments have been made to the camera, at block 104 the tip of thepen is pressed anywhere on the tablet to freeze the digital image ontothe monitor. At block 106, the user makes a determination if the imagecurrently displayed on the monitor is acceptable. If the image is notacceptable, the routine loops to block 100. If the image is acceptable,an appropriate command is entered at block 108 and the image is storedin nonvolatile memory for future viewing.

At block 110, a test is made to determine if an exit or other similarcommand has been entered by the user, i.e., if any more pictures are tobe taken. If additional pictures are to be taken, the program loops toblock 100. Otherwise, at block 111 a checksum value (described below) iscalculated by the imaging program for each (original) image that hasbeen stored. At block 112, the imaging program stores each image and itsassociated checksum value. The routine then returns to block 68 of FIG.3.

Determining Authenticity Using Checksum Values

The checksum value is an addendum to an original stored image that isused to determine its authenticity when the image is subsequentlydisplayed or printed. Those skilled in the art will recognize that thereare a number of methods of implementing such a checksum procedure. Forexample, one checksum computation is to add up the grayscale values forone of the colors, i.e., red, green, or blue, for each pixel comprisingthe image. Assuming a screen of 640 by 480 pixels and 256 colors perpixel, the checksum values would range from 0 to (640)(480)(255). Whenan image is recalled for display or to be printed, the checksum value isrecalculated. If the image has not been modified, the newly calculatedchecksum value will be equivalent to the addendum value, and the imageis certified as being unaltered. If the image has been modified, thechecksum values will vary, indicating the image has been modified. Inthis instance, an indication of the fact that the image has beenmodified may be displayed or printed with the image, if desired.

The authentication of an original image using a checksum value is idealfor situations in which physicians display before and after pictures ofa patient who has undergone cosmetic surgery. In some instances, viewersare skeptical as to whether an "after" image is really representative ofa patient's appearance after surgery. This is in reaction to beliefsthat post-surgical images have been altered to make the patients lookbetter. For example, there may be temptation to slightly fade wrinklesor otherwise edit features that the physician was attempting to addressin a surgery. Using the described checksum feature, the post-surgeryimage can be verified as an authentic, unaltered image based upon theaddendum value, and the veracity of the image is not questioned. This isbeneficial to physicians when illustrating post-surgical results duringlectures or in other teaching situations.

FIG. 4B illustrates an exemplary routine for determining whether or notan original image, i.e., pre- or post-surgical image, has been alteredin accordance with the invention. This routine may be invoked whenever apre- or post-surgical image is displayed on a monitor or printed on apage. At block 114, a test is made to determine whether the image to bedisplayed is portrayed as an "original image" that was not modified,e.g., a before or after picture. If the image is supposed to be anoriginal image, a current checksum value for the image is calculated atblock 115. At block 116, the calculated checksum value for the image iscompared to the checksum that was stored when the image was acquired bythe aesthetic imaging system, e.g., when the picture was taken.

At block 117, the calculated checksum value is compared to the storedchecksum value to see if they are equivalent. If the two values are notequivalent, at block 118 an icon is added to the image, e.g., displayedor printed along with the image, indicating that the image has beenaltered. If the checksum values are equivalent, an icon is added to theimage at block 119 verifying the authenticity of the image. Once eithericon has been added to the image, or if the image being printed ordisplayed is not an original image, the program terminates. As will beappreciated, the same checksum computational method must be used on eachimage, i.e., when an original image is acquired and when an image is tobe displayed, or the comparison will be meaningless.

While the use of a checksum is contemplated in the exemplary routine fordetermining whether an original image has been altered, it will beappreciated that other techniques may be used to detect when an originaldigital image has been modified. For example, a flag or other marker maybe uniquely associated with the original digital image. If the flag orother marker is absent from the digital image being displayed, thedigital image is a copy that is presumed to have been changed.Alternatively, two versions of the image may be stored, included anunaltered original and a copy. The two images may be compared in orderto determine whether modifications have been made to the copy.

The editing aspects of the invention are now described.

Editing Using Draw Tools

A disadvantage of prior art aesthetic imaging systems is that aphysician or facilitator may have to cycle back and forth betweenseveral menus in order to properly edit an image. In an effort tominimize the number of menus required, the imaging program includes aunique combination draw (CD) feature that generally works with all ofthe drawing tools. The CD feature allows a user to freehand draw, usecurves to edit an image, as well as undo using either freehand orcurves, without having to invoke a separate menu for each item.

The following describes an interface for the CD feature as implementedin an embodiment of the invention. For purposes of this discussion, itwill be assumed that the airbrush tool is selected as the drawing tool,although it is to be understood that the CD tool generally applies toall of the drawing tools. Upon selection of the airbrush draw tool, theaesthetic imaging system prompts the user to choose a color from a colorpalette that appears on the monitor. A color is selected using the pen.After selecting a color, a side bar menu is displayed. The user mayselect from a number of options on the side bar, including brush sizeand intensity, or select away from the side bar menu to remove the menufrom the screen. In the latter case, the system defaults are used.

To freehand draw, the user presses on the tablet with the pen tip andcontinues pressure while moving or "rubbing" the pen on the tablet. Atthis point the chosen color is written onto the image at the location onthe monitor that corresponds to the pen location. Pressing the side barwhile repeating the motion will allow the user to selectively remove anyedits to the image using a freehand motion.

To draw a curve, the user must set a first anchor point by selectingwith the pen. Thereafter, as the user moves or "floats" the pen acrossthe tablet, a green line will extend from the first anchor point to thecurrent position of the pen. In a desired location, a second anchorpoint is set by selecting with the pen. Once both anchor points havebeen established, the green line appears on the monitor as a segmentbetween the two anchor points. To create a curve, the user floats thepen across the tablet. The system will display a curved line bending andmoving with the movement of the pen. The pen movement (top to bottom, orside to side) determines the arc of the curve. As the curve moves acrossthe image, the image is edited in accordance with the selected draw tooland draw tool options. With the airbrush example, the system displaysthe curved line repeating itself with the chosen color. Pressing theside bar while repeating the motion will allow selective removal of anyedits to the image using a curve established between the anchor points.

FIG. 5 illustrates an exemplary routine for implementing the CD featureof the imaging program. In one embodiment of the invention, the drawtool group includes: Airbrush, Tint, Texture, Blend, Undo, and Contour.Once a draw tool is selected from the main menu at block 74 of FIG. 3,the routine of FIG. 5 begins and selections must be made, or defaultselections confirmed, for the selected draw tool. Specifically, at block120 the user is prompted to enter any options that are applicable to thedraw tool selected. The airbrush tool will be described as an example.With airbrush, a color must be chosen. After a color has been chosen, aside bar menu is displayed at block 122 which illustrates other optionsthat may be invoked for the airbrush tool. These typically includeselecting a brush size, brush intensity, and other miscellaneousoptions. Sample side bar menus for various shape tools are describedbelow. The draw tool side bar menus are similar to these.

Once the side bar menu options are entered, a test is made at block 124to determine if the side button has been depressed. Pressing the sidebutton can mean a cancel/back-up or an "undo," depending upon when it isactivated. It is noted that at any point during the draw routine and theother routines described below, the side button may be used to return tothe main menu through multiple backup or cancel commands.

If the side button has not been pressed, a test is made at block 126 todetermine if the tip of the pen has been pressed. If the tip has notbeen pressed, the routine loops to block 124. Otherwise, a test is madeat block 128 to determine if the tip was released prior to moving thepen. If the pen has been moved while the pen tip was pressed against thetablet, the imaging program enters freehand draw mode, shown at block130. In one embodiment of the invention, freehand draw mode is enteredif the pen moves the equivalent of three or more pixels. While infreehand draw mode, freehand edits may be made to the image in a mannersimilar to prior art imaging programs. This mode will remain untilpressure on the pen tip is released. After beginning freehand draw mode,the routine loops to block 124.

If the pen was not moved, e.g., the pen has been moved two pixels orless prior to releasing the tip, the imaging program begins a curve modeby establishing a first endpoint, as indicated at block 132, and drawinga line on the monitor from the endpoint to the current pen position,indicated at block 134. At block 136, a test is made to determine if thetip has been pressed. The imaging program at this point is looking for asecond endpoint to be entered. If not, the program loops back to block136 to await the input. As noted above, the user can go back to thebeginning of the routine using the cancel button if the user has changedhis or her mind, although this is not shown in the flow diagram.Specifically, the first cancel would place the routine at block 124, thesecond at either block 122 or 124, depending on the draw tool selected,and subsequent cancels would forward the routine to the main menu.

Once the tip is pressed, the second endpoint is established at the tipposition, and a line segment is drawn on the monitor from the twoendpoints, as indicated at block 138. At this point in the routine, theimaging program is in "curve draw mode" as indicated at block 140, andthe user can make any edits desired using a curvilinear line segmenthaving the established endpoints. The routine loops to block 124 whilein this mode.

With reference to block 124, if the side button has been depressed atest is made at block 142 to determine if the imaging program is ineither the freehand or curve draw modes. If the imaging program is ineither mode, an undo mode will be invoked as long as the side buttonremains depressed, as indicated at block 144. If at the time ofdepressing the side button the imaging program is in freehand draw mode,the undo will also be freehand. Selective freehand undo edits are thusavailable. Similarly, if at the time of depressing the side button theimaging program is in curve draw mode, the undo will be in this mode.Selective "undo" edits are then available using a curve, as opposed tofreehand drawing. Releasing the side button will return the imagingprogram to the drawing mode that was active just prior to depressing theside button.

Once the side button is released, or if the imaging program was not ineither drawing mode, a test is made at block 146 to determine if theuser wishes to exit. This test generally refers to the user againdepressing the side bar, and thus "backing out" of the drawing routine.In this case, the draw routine returns to block 78 of FIG. 3. Otherwise,the routine branches to block 124.

To further exemplify the advantages of the CD draw tool described inFIG. 5, an original (unedited) image 130 that is representative of animage displayed on a monitor is illustrated in FIG. 6. A main menu 132is displayed across the top of the image to allow a user to selectediting, viewing and printing options, as discussed in FIG. 3 andaccompanying text. The main menu 132 is from an embodiment of theaesthetic imaging system 20.

FIGS. 7A-7E and 8A-8E contrast exemplary steps taken to make identicaledits to the image 130. The steps required to modify the image using arelatively advanced prior art imaging system are illustrated in FIGS.7A-7E. These steps are modeled after a prior art imaging system that hasbeen distributed by Mirror Image Technology, Inc., a division of VirtualEyes, Incorporated, the assignee of the present invention. The stepsrequired using an embodiment of the aesthetic imaging system 20 inaccordance with the invention are illustrated in FIGS. 8A-8E. Briefly,each set of drawings illustrates examples of edits to a patient's nose,chin, and neck regions. The edits are for use in explaining theinvention only, and merely exemplary in nature.

A brief description of the edits follows. With reference to FIG. 7A, thecurve option of an airbrush tool is used to modify the bridge of apatient's nose. A resultant curve 134 is displayed having anchor points(endpoints) 136 and 138. In FIG. 7B, a freehand motion is used toeliminate a portion of the tip of the patient's nose. In FIG. 7C, anundo tool is used to replace a portion of the bridge of the patient'snose that was removed by the edits performed in FIG. 7A. In FIG. 7D, theneck area of the patient has been edited using the curve option of anairbrush tool. The resultant curve 140 has anchor points 142 and 144.Finally, in FIG. 7E, an undo tool is used to add back a portion of theneck area that was removed in FIG. 7D. These edits require the followingsteps:

    ______________________________________                                        FIG. 7A:                                                                              Step S1   move pen to draw on main menu;                                      Step S2   select draw;                                                        Step S3   move pen to airbrush;                                               Step S4   select airbrush;                                                    Step S5   move pen to curve;                                                  Step S6   select curve;                                                       Step S7   select an airbrush color;                                           Step S8   move pen to the first anchor point                                            position;                                                           Step S9   select at the position to establish the                                       anchor point 136;                                                   Step S10  move pen to the second anchor point                                           position;                                                           Step S11  select at the position to establish the                                       anchor point 138;                                                   Step S12  move pen to bend the curve 134 into the                                       bridge of the nose;                                         FIG. 7B:                                                                              Step S13  press the side button on the pen to exit                                      to the main menu;                                                   Step S14  move pen to draw;                                                   Step S15  select draw;                                                        Step S16  move pen to airbrush;                                               Step S17  select airbrush;                                                    Step S18  move pen to freehand;                                               Step S19  select freehand;                                                    Step S20  select a color for the airbrush tool;                               Step S21  use a rubbing motion with the pen to                                          make the freehand edit;                                     FIG. 7C:                                                                              Step S22  press the side button on the pen to exit                                      to the main menu;                                                   Step S23  move pen to draw;                                                   Step S24  select draw;                                                        Step S25  move pen to undo;                                                   Step S26  select undo;                                                        Step S27  move pen to curve;                                                  Step S28  select curve;                                                       Step S29  use a rubbing motion with the pen to                                          undo the previous edit;                                     FIG. 7D:                                                                              Step S30  press the side button on the pen to exit                                      to the main menu;                                                   Step S31  move pen to draw;                                                   Step S32  select draw;                                                        Step S33  move pen to airbrush;                                               Step S34  select airbrush;                                                    Step S35  move pen to curve;                                                  Step S36  select curve;                                                       Step S37  select color to be used by the airbrush                                       tool;                                                               Step S38  move pen to the first anchor point                                            position;                                                           Step S39  select at the position to establish the                                       anchor point 142;                                                   Step S40  move pen to the second anchor point                                           position;                                                           Step S41  select at the position to establish the                                       anchor point 144;                                                   Step S42  move pen to bend the curve 140 toward                                         the neck, thereby making the edit                                             shown.                                                      FIG. 7E:                                                                              Step S43  press the side button on the pen to exit                                      to the main menu;                                                   Step S44  move pen to draw;                                                   Step S45  select draw;                                                        Step S46  move pen to undo;                                                   Step S47  select undo;                                                        Step S48  move pen to freehand;                                               Step S49  select freehand;                                                    Step S50  use a rubbing motion with the pen to                                          undo a portion of the previous edit; and                            Step S51  press the side button to exit back to the                                     main menu.                                                  ______________________________________                                    

In accordance with the invention, the steps required to perform the sameedits using the aesthetic imaging system 20 are now described. Withreference to FIGS. 8A-8E, the steps required to perform the editsinclude:

    ______________________________________                                        FIG. 8A:                                                                              Step N1   move pen to draw;                                                   Step N2   select draw;                                                        Step N3   move pen to airbrush;                                               Step N4   select airbrush;                                                    Step N5   elect any color for the airbrush tool                               Step N6   move pen to the first anchor point                                            position;                                                           Step N7   select at the position to establish the                                       anchor point 136;                                                   Step N8   move pen to the second anchor point                                           position;                                                           Step N9   select at the position to establish the                                       anchor point 138;                                                   Step N10  move pen to bend the curve 134 into the                                       bridge of the nose;                                         FIG. 8B:                                                                              Step N11  pressing the tip of the pen against the                                       tablet and use a rubbing motion to make                                       the freehand edit.                                          FIG. 8C:                                                                              Step N12  pressing the tip of the pen and the side                                      button simultaneously, and maintain                                           pressure while rubbing in the area to be                                      undone;                                                     FIG. 8D:                                                                              Step N11  move pen to the first anchor point                                            position;                                                           Step N12  select at the position to establish the                                       anchor point 142;                                                   Step N13  move pen to the second anchor point                                           position;                                                           Step N14  select at the position to establish the                                       anchor point 144;                                                   Step N15  move pen to bend the curve 140 toward                                         the neck, thereby making the edit                                             shown.                                                      FIG. 8E:                                                                              Step N16  press tip of pen and the side button                                          simultaneously, while rubbing to undo a                                       portion of the previous edit; and                                   Step N17  release pressure on the pen and side                                          button, and press the side button to                                          return to the main menu.                                    ______________________________________                                    

From the simplified edits shown in FIGS. 7A-7E and FIGS. 8A-8E, it willbe apparent that the aesthetic imaging system 20 provides a distinctadvantage over prior art systems. Specifically, patients find thecontinued back and forth motions required to select necessary tools fromthe main menu to be disconcerting. The aesthetic imaging system 20simplifies the editing process by providing freehand, curve, and undooptions in the pen commands themselves, instead of in separate pull-downmenus as is done in the prior art. The combination of tools is extremelyeffective in performing edits quickly, efficiently, and nearlyseamlessly, all of which benefit both patient and physician during theconsultation process.

FIG. 9A illustrates a user interface for a contour tool for use inediting images in accordance with the invention. The contour tool isinvoked from the main menu of the imaging program, as indicated at block200. The contour tool has similarities to a blend tool, but utilizespixel manipulation to pull pixels from one area to another. For example,the tool works great for chin and lip augmentations.

After the contour tool has been selected, a side bar menu is displayedby the aesthetic imaging system, as indicated at block 201. At block202, the point size for the tool may be selected from the side bar menu.At block 203, an opacity percentage is entered by the user. If theopacity is at 100 percent, any areas affected by the edit are completelycovered by the replacement pixels. As the percentage is reduced, moreand more of the original pixels will remain creating a blending of thereplacement and prior pixels.

Anchor points are selected at block 202. The selection is accomplishedas described in blocks 128 and 130 of FIG. 5. As also described, oncethe anchor points are selected, a line is displayed between the pointsby the aesthetic imaging system, as indicated at block 205. At block204, a curve having the anchor points as endpoints is positioned along apart of the body, e.g., lips or chin, to be edited. The body part isthen edited by dragging the curve in the direction in which a body partis to be augmented. Edits made in block 206 are saved at block 208 bypressing and then releasing the tip of the pen. The program thenterminates.

FIGS. 9B-9C further describe the operation of the contour tool, byillustrating how pixels are replicated from one area of an image toanother. The image areas described are for exemplary purposes only, andare simplified for clarity in this discussion. With reference to FIG.9B, an area 209 of an image is comprised of red 218R, green 218G, blue218B, and yellow 218Y areas separated by boundary lines 210, 212, 214,and 216. It is assumed that a pair of anchor points 218 and 219 havebeen established by a user along the boundary 212, wherein the aestheticimaging system will display a line segment 220 between the two anchorpoints.

In FIG. 9C, it is assumed that the user has moved the midsection of theline segment 220 to the right. In this instance, the blue area 218B hasbeen stretched into the yellow area 218Y. This area is bounded by theline segment 220 (now curved) and the boundary line 214. Also, the greenarea 218G has been stretched into the blue area 218B. This area isbounded by a curved line segment 221 and the boundary line 212. The redarea 218R has expanded into the green area 218G; this area is bounded bya curved line segment 222 and the boundary line 210.

If the opacity level is at 100 percent, the newly defined areas will becomprised of the color being expanded. Thus, the area bounded bysegments 220 and 221 will be blue; the area bounded by segments 221 and222 will be green; and the area bounded by segment 222 to the left edgeof the diagram will be red.

If the opacity level is less than 100 percent, pixels from theunderlying image areas that are being written over by the newly definedareas will be blended into the newly defined areas. At an opacity of 80percent, for example, the area bounded by segments 220 and 221 willstill be primarily blue, but the portion of this area bounded by thesegment 220 and the boundary line 214 may have a yellow tinge; and theportion of this area bounded by the boundary line 212 and segment 221may have a green tinge. As the opacity percentage is dropped, theeffects on these areas will be even greater.

While somewhat simplistic, the illustration in FIGS. 9B and 9C describesthe function of the contour tool and the effect on more complicatedpixels patterns will be appreciated by those skilled in the art. Thearea bounded by segment 220, boundary line 214, segment 221, andboundary line 212 will not be affected by changes in the opacitysetting.

FIGS. 9D-9G illustrate edits to a patient's lips 224 using the contourtool. FIG. 9D is a "before" picture without any modifications. To editthe right half of the patient's upper lip, the user first selects thedesired end points surrounding the feature to be modified. A firstanchor point 374 is designated near the middle of the upper lip, and asecond anchor point 376 is designated at the right outer margin of theupper lip. When the anchor points have been selected, a line 378 isdisplayed between the points by the aesthetic imaging system. Byfloating the pen over the tablet in a direction generally indicated byarrow 380, the line is bent to form a curve 382 that approximates thecontour of the feature being edited. When the user has fitted the curveto the feature, the shape of the curve is set by tipping the pen. Asshown in FIG. 9E, the user then selects a point along the portion of thecurve by tipping the pen at the desired location. By floating the penabove the tablet, the user can stretch the selected feature in themanner described above. In FIG. 9E, the user floats the pen in adirection generally indicated by arrow 384 to enlarge the patient'sright upper lip. As the upper lip is expanded, the pixels forming theupper lip are replicated to expand the portion of the lip, while thepixels outside of the upper lip are deleted.

In FIG. 9E, the left side of the person's upper lip has been editedusing the contour tool. In FIG. 9F, both the left and right sides of theperson's upper lip are shown modified using the contour tool. In FIG.9G, the lower lip has been edited. Using the contour tool, these editsare accomplished very quickly, in part because the augmented area willautomatically match the area around it. The features of the contour toolmake it much easier to perform augmentations than is currently availableusing prior art imaging tools.

Editing Using Shape Tools

FIGS. 10-12B are directed to the shape tool features of the aestheticimaging system. Similar to the draw tool, the imaging program has acombination feature that is generally available with any of the shapetools. This feature, called the autoblend feature, allows the user toeasily move and paste shape tool edits, with or without blending theedges around the edit.

FIG. 10 illustrates an exemplary routine for implementing the autoblendfeature of the imaging program. The shape tool group generally includes:Warp, Stretch, Copy, Cutout, Rotate, Freeze Compare, and Resize. Theroutine of FIG. 10 is invoked after a shape tool has been selected atblock 80 of FIG. 3. The side bar menu for the selected shape tool isdisplayed at block 230. Exemplary side bar menus for various shape toolsare set forth below. The side bar menus are illustrated in an effort tofurther detail options available for a given tool, and may be especiallyhelpful for those unfamiliar with imaging packages.

    ______________________________________                                        Copy, Cutout, ResizeStretch, RotateFreeze Compare                             ______________________________________                                         ##STR1##                                                                     ______________________________________                                    

The black dots are brush size options that allow a user to choose thethickness of a shaping tool. The zoom option allows a user to look at animage in greater detail. When the zoom option is invoked, the aestheticimaging system displays a square overlaid on the image. The square canbe positioned by the user with the pen. After positioning the square,that area of the image will be magnified when the pen is selected.Canceling with the pen will display normal viewing mode. The undo optionallows a user to undo edits to an image. The compare option allows auser to transition between before and after images. The split imageoption allows a user view before and after images side by side. Theinverse option creates the mirror image of all or only a portion of animage that has been designated by the user. The blend tool will blendedits with the surrounding area. Many of the options shown are alsoimplemented as separate tools under View in the main menu. These aredescribed in greater detail below.

It is noted that the side bar menus available for the drawing tools aresimilar to the shape tool side bar menus shown. They do, however,typically include an option wherein the user may choose the intensity oropacity of a color used in conjunction with a draw tool.

At block 232, the user is prompted to designate an area of the image tobe edited. In a preferred embodiment, this is accomplished by pressingdown on the pen and freehand drawing an area, e.g., a circle, that is tobe subject to the edit. In this regard, the imaging program contains aunique feature wherein if a partial area is designated and the pensubsequently released, the drawing area will automatically be formedinto a contiguous area by the imaging program. At block 236, a test ismade to determine if an area has been designated by the user. If not,the routine loops back to block 234 and awaits a designation.

After an area has been designated, any edits to the designated area ofthe image are performed in accordance with the selected shape tool, asindicated at block 236. Two exemplary shape tools for editing an imageare illustrated in FIGS. 12A and 12B. At block 238, a test is made todetermine if editing of the designated area is complete. In oneembodiment, this involves testing for when the user "selects" with thepen anywhere on the tablet. The routine remains at block 238 untilediting is complete (or the user exits using the side button). Uponcompletion of the edits, an autoblend box is displayed in the vicinityof the edited area, as indicated at block 240.

At block 242 a test is made to determine if the tip of the pen has beenpressed against the tablet. If not, the routine loops, testing for thisoccurrence (or an exit command from the user). After the pen tip hasbeen pressed, the imaging program calculates the location of the penrelative to the autoblend box.

FIG. 11 illustrates an example of an autoblend box 250 that may be drawnby the aesthetic imaging system in accordance with the invention. As isdiscussed above, the autoblend box 250 may be used to: (1) move anedited area, (2) paste the edited area onto the image while blending theedge created between the edited area and the rest of the image, and (3)paste the edited image without blending. While the autoblend box 250uses the conventions set forth below, those skilled in the art willappreciate that other conventions may be used without departing from thescope of the invention.

To move an edited area, the user must press down on the pen anywhereinside the autoblend box, except not at the three-, six-, nine-, andtwelve-o'clock positions of the box. The "move" area is designated withreference numeral 254. To paste the edited area without blending, theuser must press within the outlined areas 254 at the three-, six-,nine-, and twelve-o'clock positions inside the box. Selecting anywhereoutside the box, i.e., area 256, results in the edited area being pastedwith blending.

With reference again to FIG. 10, a test is made at block 244 todetermine if the pen tip has been pressed at a location outside of theautoblend box. If the pen tip was pressed at a location outside theautoblend box, the edit area is pasted with a blending of the edges atblock 260. Otherwise, a test is made at block 262 to determine if thetip location was within the approximate three-, six-, nine-, andtwelve-o'clock areas of the autoblend box. A pressing of the tip withinany of these areas results in the edited area being pasted withoutblending, as indicated at block 264. As indicated at block 266, aselection in a location in the autoblend box apart from the three-,six-, nine-, and twelve-o'clock areas will allow the image to be moved.In this case, the edited area will track movement of the pen as long asthe tip remains pressed. After a move is completed, the routine loops toblock 242.

If a paste has been accomplished using blocks 260 or 264, a test is madeat block 268 to determine if the user wishes to exit the shape routine,e.g., by pressing the side button. If not, the routine loops to block232 where a new area of the image may be considered. Otherwise, theroutine returns to block 82 of FIG. 3.

FIGS. 12A and 12B illustrate two exemplary shape tools that areavailable when using the aesthetic imaging system. With reference toFIG. 12A, a cutout tool is unique in that a user can select an area ofthe image to be cut out, thereby creating a "hole" in the image, and anidentical image underneath the cutout image can then be moved in alldirections as it is viewed through the hole. The cutout feature isespecially useful for profile views including chin augmentation, browlifts, and maxillary and mandibular movement; and frontal views,including otoplasty, brow lift, lip augmentation, nasal base narrowing,and maxillary and mandibular movement.

At block 270, the current image is copied to a working buffer. As isdiscussed in FIG. 10, when the cutout subroutine is called the user hasdefined an area of the image to be edited. At block 272, a boundary iscreated around the designated area designated in block 234 of FIG. 10.At block 274, the working area is displayed inside the boundary, and thecurrent image displayed outside the boundary. In this manner, the imagein the working buffer can be moved relative to the image in the currentbuffer until the desired alignment has been achieved. The program thenreturns to the routine of FIG. 10.

Upon returning to block 238, the edit may be frozen by selecting withthe pen. Thereafter, the autoblend box is displayed. Selecting withinthe area 252 of the autoblend box allows the designated area to bemoved. Selecting anywhere outside the autoblend box will make the editpermanent, with automatic blending. Selecting within the box at thethree-, six-, nine- or twelve-o'clock positions (areas 254) will makethe edit permanent, without blending.

With reference to FIG. 12B, a rotate tool in accordance with theinvention is particularly useful when editing profile views includingthe nasal tip, mandible, maxilla and brow areas; and frontal viewsincluding nares, brows, and the corners of the mouth. As is discussed inFIG. 10, when the rotate routine is called, the user has defined an areaof the image to be edited. At block 276, the area designated in block234 of FIG. 10 is shown in phantom. At block 278, the imaging programwaits for the user to enter an axis of rotation. An axis is then enteredby the user, as indicated at block 279.

Once an axis of rotation is entered, a display line emanating from theaxis point is displayed on the monitor, as indicated at block 280. Also,the number of degrees of rotation is displayed. The position of the pendictates the degree of rotation. As the pen is moved away from the axispoint, the display line will lengthen, providing the user greatercontrol of the rotation of the designated area. At block 282, the systemwaits for the user to enter a desired degree of rotation. The degree ofrotation is entered by the user by selecting with the pen, as indicatedat block 283. Once the degree of rotation is entered, the designatedarea is redrawn onto the current image, as indicated at block 284.

After the redraw, the routine returns to FIG. 10. Upon returning, theautoblend box is displayed. Selecting within the autoblend box allowsthe designated area to be moved. Selecting anywhere outside theautoblend box will make the edit permanent, with automatic blending.Selecting within the box at the three-, six-, nine-, and twelve-o'clockpositions will make the edit permanent without blending.

While prior art imaging programs have a rotate feature, they do notallow a user to select the axis of rotation. The ability to select theaxis is valuable in the procedures listed above.

FIGS. 18A-18D disclose the capabilities of the warp shape tool in theaesthetic imaging system. The warp shape tool is a powerful tool thatallows users to easily edit a patient's features. Similar to the contourtool, the warp tool allows a user to select and manipulate a feature ofthe patient's image, with the aesthetic imaging system automaticallyredrawing the area surrounding the manipulated feature as themanipulation is being performed.

As shown in FIG. 18A, the user first defines an image to be manipulatedby the warp tool by encircling the portion of the image to be edited asshown by dotted line 386. Once the area has been selected, a user maytip the pen to designate a stretch point within the selected area. Asshown in FIG. 18A, a stretch point 388 has been designated near the topportion of the patient's right upper lip. Once the stretch point hasbeen designated, the user may float the pen in the desired directionthat they would like to stretch the image. The portion of the image thatis located at the stretch point is pulled in the direction that the userfloats the pen, with the area surrounding the stretch point eitherexpanded or compressed. That is, the area in the direction that the userfloats the pen is compressed, and the area away from the direction thatthe user floats the pen is expanded. The amount of expansion orcompression is dictated by the distance of the area away from thestretch point, in a manner discussed in greater detail below.Preferably, the image is manipulated in real time, so that the user ispresented with a seamless and continuous stretching or movement of theselected feature.

The technique used to implement the warp tool is portrayed in FIGS.19A-19E. A user first defines a manipulation area 420 to be edited bycircling the desired feature of the patient. As shown in enlarged detailin FIG. 19B, when the manipulation area has been selected, the aestheticimaging system creates a pixel map of a warping area 422 that completelyencompasses the manipulation area. For computational purposes, thewarping area is preferably rectangular and approximately 33% larger inarea that the manipulation area. It will be appreciated, however, thatthe warping area may be differently shaped or sized depending on theparticular application and available system hardware.

After defining the manipulation area, the user selects a stretch point424 within the manipulation area by tipping the pen at a desiredlocation within the area. As shown in FIG. 19C, when the stretch pointis selected, the aesthetic imaging system maps four rectangles 426a,426b, 426c, and 426d in the warping area. One corner of each rectangleis defined by the stretch point, and the diagonally opposing corner isdefined by a corner of the warping area. Perpendicular lines 427 drawnthrough the stretch point intersect the warping area boundary at points428a, 428b, 428c, and 428d. After the stretch point is selected, theuser may float their pen to manipulate the selected feature, for examplein the general direction indicated by arrow 430.

As the user floats the pen to warp the selected feature, the rectanglesin the warping area are distorted. In FIG. 19D, the stretch point hasbeen moved upwards and to the left in the warping area, to anintermediate location 432. Periodically during the stretch point'smotion, the aesthetic imaging system determines the shape of fourquadrilaterals 434a, 434b, 434c, and 434d, with sides that extend fromthe intermediate location 432 of the stretch point to original points428a, 428b, 428c, and 428d. Using a bilateral transformation, the pixelsin the original rectangles 426a, 426b, 426c, and 426d are then mappedinto the quadrilaterals. Further movement of the stretch point repeatsthe process, the original set of rectangles being mapped into the newquadrilaterals that are generated by the movement. If the display isupdated frequently as the stretch point is moved, the user views themapping and remapping as a warping of the patient's feature within themanipulation area. It will be appreciated that only the portion of thewarping area that coincides with the manipulation area is presented tothe user. The pixels comprising the remaining portion of the warpingarea are used for computational purposes, but are not displayed to theuser.

When the desired warping effect is achieved, the user tips the pen tofix the stretch point in a desired location. To ensure there is nodiscontinuity between the manipulation area and the surrounding area ofthe patient's image, automatic blending around the outer margins of themanipulation area is performed by the aesthetic imaging system. As shownin FIG. 19E, the aesthetic imaging system also remaps the warping area,creating four new rectangles 436a, 436b, 436c, and 436d based on theending location 438 of the stretch point. In a preferred embodiment ofthe invention, a second stretch point may then be selected within themanipulation area and the process repeated, with the second warpingtransforming rectangles 436a, 436b, 436c, and 436d that resulted fromthe first warp. After the second warping is completed, the user may thenselect a third or additional warp point to further manipulate the image.Each manipulation is performed without the user having to redefine themanipulation area.

The flexibility and power of allowing multiple warps on a selectedregion is demonstrated in FIGS. 18A-18C. Returning to FIG. 18A, a userdesignates a stretch point 388 and floats the pen in a directiongenerally indicated by arrow 390. Floating the pen upwards and outwardsgenerally causes the upper right portion of the lip to be expanded inthat direction. The resulting manipulation is shown in FIG. 18B, whereinthe right lip has been made fuller. The area surrounding the lip isexpanded or compressed to ensure that there are no discontinuitiesbetween the edited lip and the surrounding face. In FIG. 18B, the userselects a second stretch point within the manipulation area to furtherstretch the selected feature. For example, a stretch point 392 may bepositioned on the upper portion of the left lip, and the pen floated ina direction generally designated by the arrow 394. As shown in FIG. 18C,this generally causes the upper left portion of the lip to be madefuller to match the upper right portion of the lip. Again, during thewarping the aesthetic imaging system automatically expands or contractsthe surrounding unmanipulated area to ensure that there are nodiscontinuities between the upper left and the unedited portion of theface.

The warp tool with multiple stretch points is a very powerful tool as itallows the user to quickly manipulate an image with a minimum use ofdrawing tools or piecemeal editing. Because the warp tool performs themanipulation in real-time, the edits are accomplished very quickly andfluidly. A user may therefore generate a desired image in a minimalamount of time.

Viewing an Image

FIG. 13 illustrates an exemplary routine for implementing the viewfeatures of the imaging program. In FIG. 13, the solid blocks indicateuser interface options presented to the user by the aesthetic imagingsystem and the dashed blocks represent system responses to the decisionsmade. The view group includes: Compare, Prioritize, Split Image, MirrorImage, and Restore to Original, as well as other options including Zoomand Emboss. At block 300, a test is made to determine if the Compareoption has been selected.

The Compare option allows a modified image to be compared to theoriginal image so that a viewer can more readily see the changes.Specifically, as the pen is floated from the top to the bottom of thetablet, the user will see one image transition or "morph" into theother. The morphing is accomplished by overlaying the original imagewith the modified image, and varying the opacity of the modified image.When the modified image is opaque, only the modified image may be viewedby a user. When the modified image is completely transparent, only theoriginal image may be viewed by a user. In between these two extremes,varying amounts of the edits made to the image will be apparent to theuser. The feedback to the patient as the original image morphs into themodified image is much more powerful than a side-by-side comparison ofthe two images.

When a desired comparison level is achieved, a user can press the tip ofthe pen to freeze an image displayed on the monitor at a point anywherefrom zero to 100% of the transition from the original to the modifiedimage. Freezing an image at a partial transition is extremely helpfulwhere edits have been performed on an image that are not realisticallyachievable in surgery, but an achievable result lies somewhere betweenthe original and the modified image. For example, it is easy to edit ablemish on a face so that area resembles the surrounding skin and thusbecomes invisible. However, the total removal of the blemish may not berealistic. In this case, a transition of that area toward the originalimage will slowly "fade in" the blemish. A physician may then freeze thefading process at a desired point to provide a realistic image of whatsurgery can achieve to the patient.

Another option that is a subset of the Compare option is the Prioritizeoption. The Prioritize option allows a user to designate areas that havebeen edited so that the user can selectively illustrate the effects oftwo or more procedures that have been shown to a patient. For example,FIG. 14A illustrates a modified profile image 302 of a patient thatincludes a rhinoplasty procedure (nose) 304, a chin augmentationprocedure 306 and a submental lipectomy procedure (neck) 308. Theboundaries that have been edited are illustrated by dashed lines 304a,306a, and 308a, corresponding to the patients original nose profile,chin, and neck, respectively. Using the Prioritize option, the user candesignate one area on the modified image, and illustrate transitionsbetween the original and modified images at that area only by floatingthe pen. Any areas not selected will continue to be displayed as theoriginal image.

With reference again to FIG. 13, if the Compare option has beenselected, a test is made at block 309 to determine if the entire imageis to be compared or only certain portions of the image, i.e., using thePrioritize option. If less than the entire image is to be compared, theuser is prompted to enter the area or areas that are to be compared atblock 310. A user may then define one or more "priority areas" byfreehand circling the desired area. When the priority areas or defined,or if all of the edits are to be reviewed during the comparison, atblock 311 the user is prompted to float the pen in a vertical motion ontop of the tablet to transition between the original and modifiedimages, in accordance with the Compare feature discussed above. Anillustration will clarify this point.

In FIG. 14B, a first priority area 312 has been defined that correspondsgenerally to the nose. Given this selection, the nose area only willtransition from original to modified as the pen is moved, with the restof the image being displayed unedited. Thus, the modifications to thechin and neck no longer are shown. In FIG. 14C, a second priority area314 has been defined that corresponds generally to the chin. The firstpriority area 312 has been kept. Given these selections, the nose andchin areas only will transition from original to modified as the pen ismoved, with the rest of the image being displayed unedited. Thus, themodifications to the neck are not illustrated.

With reference to FIG. 14D, a third priority area 316 has been definedthat corresponds generally to the neck, along with the formerdesignations. Given these selections, all three priority areas 312, 314,and 316 transition with movement of the pen. Again, the undesignatedportions of the image are displayed in an unedited form, even if partsof the image outside the priority areas have been edited (no edits areshown). Because edits have not been made, from a user's standpoint thetransition in FIG. 14D appears to be a comparison between the originaland modified images.

While preferably the user defines the priority areas on the image beingedited, it will be appreciated that the priority areas may also beautomatically defined by the aesthetic imaging system. A comparison maybe made between an original image stored in a buffer and the editedimage that has been modified by the user. Any areas containingdifferences between the original and the edited image may be highlightedby the aesthetic imaging system, and a priority area automaticallydefined for each area containing differences. Whether a priority area isdefined may also depend on the number of differences between theoriginal image and the edited image.

In an alternative embodiment of the invention, when a user defines oneof the priority areas in the Prioritize option, the user is prompted toenter a textual identifier for the defined area. For example, aftercircling the area in FIG. 14B and defining a first priority area 312,the user would be prompted to enter a textual identifier correspondingto the first priority area. The user may assign a descriptive identifierrelated to that priority area, such as "nose", or the user may assign anon-related textual identifier such as "area1". Textual identifiersassigned by the user in this manner are displayed to the user in asubmenu of the Prioritize option. The user may then select from thesubmenu of the Prioritize option those areas that they wish to displayfrom the list presented to them. For example, the prioritize submenucorresponding to FIG. 14D may present the user with a choice of "nose,""chin," and "neck." By selecting the nose and neck from the prioritizesubmenu, the user could simultaneously display only the effects of thenose and neck procedures. After making such a comparison, the user coulddeselect neck from the prioritize submenu, and instead select the chinand nose. This would allow the user to display only the effects of thechin and nose procedures using the compare option. It will beappreciated that assigning textual identifiers to each of the definedpriority areas provides greater flexibility to a user, since the userdoes not have to redefine each of the priority areas that are to bedisplayed each time. A user may therefore quickly cycle through variouspermutations of the procedures that have been edited for patientdisplay.

In the alternative embodiment in which textual identifiers are assignedto each of the priority areas, it will be appreciated that variousmethods can be used to display to the user the correspondence betweenthe textual identifier and the priority area. For example, when the userselects one of the textual identifiers, the priority area could behighlighted or otherwise encircled with a contrasting color to indicateto the user the area of the image that corresponds to that textualidentifier. Similarly, a user may also point to specific areas of thedisplayed image and have the textual identifier corresponding to thatarea appear to the user. Displaying the correspondence between priorityarea and identifier would allow a user to rapidly determine theavailable priority areas that may be shown to a patient when the userhas not recently worked with or otherwise viewed the image.

With reference again to FIG. 13, at block 320 a test is made todetermine if the user wishes to save a transitional or morphed view ofan image. If a transitional view is to be saved, the user may establishthe percentage transition, i.e., anywhere from zero to 100 percenttransition (zero percent being the original image and 100 percent beingthe edited image), by floating the pen up or down above the tablet toestablish the view, and the pressing the tip of the pen against thetablet to freeze the transitional image, as indicated in block 322. Ifthe tip is pressed again, the frozen image is saved. The save optionsare available with or without the priority areas in effect. After thesave has been accomplished, or if the user did not wish to save atransitional view of an image, the Compare option is complete and theroutine branches to block 326.

In an alternative embodiment of the invention, the Compare option allowsa user to compare a modified image with any edits made to the modifiedimage during the current editing session, i.e., before the changes arepermanently saved to the modified image. Specifically, with reference toFIG. 2, this embodiment of the Compare option contrasts the image in thecurrent image buffer 54 with the image in the modified image buffer 52.As discussed above, this embodiment of the Compare option may also beused in conjunction with the Prioritize option to allow the user toselect priority areas for comparison. In this case, the priority areastransition from the modified to the current image, while the modifiedimage only is displayed in the other (nonselected areas) areas.

At block 326, a test is made to determine whether the user wishes toview a split image. The Split Image option is used on a frontal pictureonly, and allows a patient to see his or her asymmetries. If a splitimage view is desired, the user is prompted to select an image, e.g.,original or modified, at block 330. At block 332, a vertical centerlineis displayed on top of the selected image. The user is then prompted toposition the centerline at the location desired, as indicated at block334. Typically, the centerline will be positioned to dissect the imageinto equal halves, using the nose and the eyes as reference points. Atblock 336, the aesthetic imaging system displays two images, one showingthe left halves pieced together and the other the right halves piecedtogether. Specifically, the aesthetic imaging system will produce aninverse image of the left (right) half and then add it to the left(right) half.

FIGS. 15A-15C illustrate the resultant images that are displayed whenthe Split Image option is invoked. In FIG. 15A, a frontal image 350 of apatient is shown, including a centerline 352 that has been positioned atthe center of the patient by a user. FIG. 15B is an illustration of theleft halves of the image after being pieced together by the aestheticimaging system, as indicated by reference numeral 354. FIG. 15C is anillustration of the right halves of the image, as indicated by referencenumeral 356. With the Split Image option, patients can view what theywould look like if their faces were symmetrical. The tool is especiallyuseful in the consultation stage because many people do not realize thatthe typical face is asymmetrical, and changing a face to be perfectlysymmetrical, if possible, is not necessarily desirable.

Once the split images have been displayed, or if the Split Image optionwas not selected, a test is made at block 360 to determine if the userwishes to view an inverse or mirror image of a picture, e.g., to showpatients the view they see of themselves when looking into a mirror. Ifan inverse image is desired, the user is prompted to select an image tobe viewed at block 362. The selected image is then "flipped" using theaesthetic imaging system, and displayed on the monitor, as shown atblock 364. After the image has been displayed, or if the inverse imageoption was not invoked, routine returns to block 86 of FIG. 3.

When viewing an image on the monitor, it is important that a patientremains focused on the image being manipulated rather than on thefeatures of the aesthetic imaging system. When certain viewing optionsare selected, for example, the prioritize option shown in FIGS. 14A-14D,or the split image option shown in FIGS. 15A-15C, the menu bar normallylocated across the top of the display is therefore removed. Only theimage of the patient is kept on the screen, ensuring that the patientremains focused on the image being manipulated.

In certain situations where it is necessary to display a menu bar,however, it is advantageous to minimize the visual appearance of themenu bar. FIG. 20 is a representative image 410 of a patient with a menubar 412 located across the top of the screen 414. To reduce thedistraction caused by the menu bar, the menu bar is preferablytranslucent to allow the user to view the patient's image through themenu bar. The text 416 and the line 418 indicative of the menu bar arepreferably presented in a contrasting, yet muted, color to allow theuser to read the commands. For example, the text and the line outliningthe menu bar may be presented in an off-white. While editing the imagewith a patient present, the patient is therefore not overly distractedwhen the menu bar periodically appears at the top of the screen. Evenwhen the menu bar is present for extended periods of time, a patient isnot distracted since it does not visually stand-out from the patient'simage. At the same time, a skilled user is provided with sufficientinformation about the menu choices to allow them to choose theappropriate menu options.

To further reduce the distraction caused by the menu bar, preferably themenu bar 412 may be moved by a user to different locations on the screen414. For example, the menu bar may be moved by a user to the bottom ofthe screen. Particularly when editing the face of a patient on thescreen, the majority of the editing will take place on the uppertwo-thirds of the screen. Locating the menu bar on the bottom of thescreen therefore positions the menu bar away from the area on which thepatient should remain focused. It will be appreciated that techniquesfor moving a menu bar to various locations on a screen are known tothose skilled in the art of designing user interfaces.

FIG. 21 is a flow chart of an exemplary routine for implementing a zoomfeature in the aesthetic imaging program. The zoom feature allows a userto increase the scale of the image to better view a selected area and toimprove the ability of the user to edit fine details in the image. At ablock 400, a zoom point is selected by a user. The zoom point identifiesthe center of the image to be expanded under the control of the user. Ata block 402, the picture is adjusted to position the zoom point at thecenter of the monitor. Centering the picture ensures that as the imageis enlarged, the portion of the image surrounding the zoom point will bedisplayed. At a block 404, the user is allowed to input a desiredmagnification factor. Preferably, the magnification factor is selectedby floating the pen from the bottom (minimum magnification) to the top(maximum magnification) of the tablet. As the user floats the pen overthe tablet, the image on the monitor is correspondingly magnified andredisplayed at a block 406. With each redisplay of the image, at a block408 a test is made to see if the user has frozen the image by pressingor tipping the pen. Once the image is frozen at a desired magnification,a user can manipulate the image using the array of drawing toolsdescribed above. It will be appreciated that for very fine work, such asremoving small wrinkles surrounding a patient's eyes, the ability tomagnify the image increases the quality of the editing that may beachieved.

Several refinements of the zoom feature may be incorporated in theaesthetic imaging program to improve the results of the zoom. Forexample, a smoothing function may be incorporated in the zoom feature toensure that as the image is magnified it does not become "pixelly" orgrainy. The smoothing function may be implemented in software.Preferably, however, the smoothing function is implemented in hardware,such as a smoothing feature provided in the Targa 2000 board describedabove and incorporated in the aesthetic imaging system. Further,feedback may be provided to the user in the form of a numerical displayon the image to indicate the approximate magnification as the userfloats the pen from the bottom to the top of the tablet. Other means canalso be implemented to allow the user to select the desiredmagnification, including a pull-down menu or numerical entry.

Yet another view option provided to the user in the aesthetic imagingsystem is an Emboss option. It has been found that the emboss viewingoption is very helpful in allowing a user to discern wrinkles or otherskin imperfections in a displayed image. By selecting the emboss option,the user causes an image of the patient to be displayed in a gray scale.The emboss option displays an image that is similar to an etching madeof a three-dimensional raised surface. A two dimensional image isportrayed, with the depth of the raised surface indicated by a darkershade of gray. The emboss option removes any deceptive informationconveyed by the color or shading of the skin of the patient and allowsany raised or depressed areas to be clearly highlighted. When viewingwrinkles or other imperfections on a patient, the emboss optiontherefore clearly identifies the raised or depressed features over thesmooth skin of the patient.

Preferably, the emboss option is implemented using a function providedin the Targa 2000 board described above and incorporated in theaesthetic imaging system. The particular function used to manipulate abitmap contained in the Targa board buffer is:

    setResizerToEmboss(const BlitListSize &size, int dx, int dy, int scale=0×100)

where:

dx and dy=the offset of the emboss operation; and

scale=a "pressure" of the emboss.

After selecting the emboss option, the user may manipulate the offset ofthe emboss by floating the pen over the tablet. Floating the penparallel to the x-axis of the tablet adjusts the dx parameter in theemboss function, and floating the pen parallel to the y-axis adjusts thedy parameter in the emboss function. It has been found that variousfeatures on the skin of a patient can be brought into greater clarity byadjusting the offsets of the embossed image. A user may therefore focuson the desired feature that they wish to show a patient. When thedesired offsets are selected, the user may freeze the offsets by tippingthe pen on the tablet. A user may then adjust the scale parameter byfloating the pen parallel to the x-axis of the tablet. Again, it hasbeen found that various skin features can be brought into greaterclarity by adjusting the pressure of the emboss. When the desiredpressure results are achieved, the user may freeze the image by tippingthe pen on the tablet.

While the preferred embodiment of the invention has been illustrated anddescribed, it will be appreciated that various changes can be madetherein without departing from the spirit and scope of the invention.For example, the advantages of the aesthetic imaging system are notlimited to imaging for cosmetic surgery only, but are applicable to anumber of medical imaging fields, including endoscopy, pathology, andhair restoration. Those skilled in the art will recognize adaptationsthat may be made to accommodate these medical imaging fields withoutdeparting from the scope of the invention. Also, a different style ofpointing device may be used in lieu of the pen and tablet.

The embodiments of the invention in which an exclusive property orprivilege is claimed are defined as follows:
 1. In an aesthetic imagingsystem for use in imaging patients for cosmetic surgery, a method ofmanipulating a digital image of a patient comprised of a plurality ofpixels, the aesthetic imaging system including a processor, a memory, amonitor, and cursor control means for moving a cursor displayed on themonitor, the method comprising:(a) allowing a user to select amanipulation area in a digital image by encircling the manipulation areausing the cursor control means; (b) defining a warping area thatencompasses the manipulation area selected by the user; (c) storing acopy of the warping area in memory as an original area; (d) allowing theuser to identify a stretch point that is contained within themanipulation area using the cursor control means; (e) allowing the userto move the stretch point from an initial position through a pluralityof intermediate positions to a final position within the warping areausing the cursor control means, the movement of the stretch pointdefining a stretch direction from the initial position to the finalposition; (f) repeatedly mapping the original area that is stored inmemory to the warping area to reflect the movement of the stretch pointwithin the warping area from the initial position to the final position,the mapping generally compressing an area of the image in the stretchdirection and generally expanding an area of the image away from thestretch direction; and (g) displaying the portion of the warping areathat is within the manipulation area selected by the user to reflect thestretching of the image caused by movement of the stretch point.
 2. Themethod of claim 1, wherein the warping area is a rectangle.
 3. Themethod of claim 2, further comprising:(a) dividing the original areainto a plurality of original quadrilaterals; (b) repeatedly dividing thewarping area into a plurality of warped quadrilaterals as the stretchpoint is moved through the plurality of intermediate positions to thefinal position, each of the plurality of warped quadrilateralscorresponding to one of the plurality of original quadrilaterals; and(c) mapping a set of pixels from each of the plurality of originalquadrilaterals to the corresponding one of the plurality of warpedquadrilaterals using a bilateral transform in order to display astretching of the digital image in real-time.
 4. The method of claim 1,wherein the mapping of the original area that is stored in memory to thewarping area is performed using a bilateral transform.
 5. The method ofclaim 1, further comprising allowing a user to sequentially identify aplurality of stretch points within the manipulation area, for each ofthe plurality of stretch points repeating steps (e) through (g) tomanipulate the image in a desired manner.
 6. The method of claim 5,further comprising storing a copy of the warping area as the originalarea following the mapping to the warping area at the final position ofeach of the plurality of stretch points, so that a subsequentmanipulation of the image will be performed on the results of animmediately preceding manipulation of the image.